Patient and Caregiver Perceptions of an Interface Design to Communicate Artificial Intelligence-Based Prognosis for Patients With Advanced Solid Tumors.

PURPOSE: Use of artificial intelligence (AI) in cancer care is increasing. What remains unclear is how best to design patient-facing systems that communicate AI output. With oncologist input, we designed an interface that presents patient-specific, machine learning-based 6-month survival prognosis information designed to aid oncology providers in preparing for and discussing prognosis with patients with advanced solid tumors and their caregivers. The primary purpose of this study was to assess patient and caregiver perceptions and identify enhancements of the interface for communicating 6-month survival and other prognosis information when making treatment decisions concerning anticancer and supportive therapy. METHODS: This qualitative study included interviews and focus groups conducted between November and December 2022. Purposive sampling was used to recruit former patients with cancer and/or former caregivers of patients with cancer who had participated in cancer treatment decisions from Utah or elsewhere in the United States. Categories and themes related to perceptions of the interface were identified. RESULTS: We received feedback from 20 participants during eight individual interviews and two focus groups, including four cancer survivors, 13 caregivers, and three representing both. Overall, most participants expressed positive perceptions about the tool and identified its value for supporting decision making, feeling less alone, and supporting communication among oncologists, patients, and their caregivers. Participants identified areas for improvement and implementation considerations, particularly that oncologists should share the tool and guide discussions about prognosis with patients who want to receive the information. CONCLUSION: This study revealed important patient and caregiver perceptions of and enhancements for the proposed interface. Originally designed with input from oncology providers, patient and caregiver participants identified additional interface design recommendations and implementation considerations to support communication about prognosis.

Design of an interface to communicate artificial intelligence-based prognosis for patients with advanced solid tumors: a user-centered approach.

OBJECTIVES: To design an interface to support communication of machine learning (ML)-based prognosis for patients with advanced solid tumors, incorporating oncologists' needs and feedback throughout design. MATERIALS AND METHODS: Using an interdisciplinary user-centered design approach, we performed 5 rounds of iterative design to refine an interface, involving expert review based on usability heuristics, input from a color-blind adult, and 13 individual semi-structured interviews with oncologists. Individual interviews included patient vignettes and a series of interfaces populated with representative patient data and predicted survival for each treatment decision point when a new line of therapy (LoT) was being considered. Ongoing feedback informed design decisions, and directed qualitative content analysis of interview transcripts was used to evaluate usability and identify enhancement requirements. RESULTS: Design processes resulted in an interface with 7 sections, each addressing user-focused questions, supporting oncologists to "tell a story" as they discuss prognosis during a clinical encounter. The iteratively enhanced interface both triggered and reflected design decisions relevant when attempting to communicate ML-based prognosis, and exposed misassumptions. Clinicians requested enhancements that emphasized interpretability over explainability. Qualitative findings confirmed that previously identified issues were resolved and clarified necessary enhancements (eg, use months not days) and concerns about usability and trust (eg, address LoT received elsewhere). Appropriate use should be in the context of a conversation with an oncologist. CONCLUSION: User-centered design, ongoing clinical input, and a visualization to communicate ML-related outcomes are important elements for designing any decision support tool enabled by artificial intelligence, particularly when communicating prognosis risk.

External Validation of a Machine Learning Model to Predict 6-Month Mortality for Patients With Advanced Solid Tumors.

This prognostic study performed external validation of a machine learning model to predict 6-month mortality among patients with advanced solid tumors.

Utility of Poison Control Center Data for Automated Opioid Overdose Surveillance.

CONTEXT: Overdosing on opioids is a national epidemic and the number one cause of death from unintentional injury in the United States. Poison control centers (PCCs) may be a source of timely data that can track opioid exposure cases, identify clusters of opioid exposure cases by geographic region, and capture opioid exposure cases that may not seek medical attention from health care facilities. OBJECTIVE: The objectives were to (a) identify data requirements for opioid overdose case ascertainment and classification and visualization in a dashboard, and (b) assess the availability and quality of the relevant PCC data for state-based opioid overdose surveillance. DESIGN: We identified types of opioid exposure, demographic characteristics, and other features that may be relevant for public health officials to monitor and respond to opioid overdose events in the community. We operationalized case definitions for an opioid overdose event based on the Centers for Disease Control and Prevention case classification definitions. We assessed the PCC database for concepts and metrics needed to operationalize case definitions for opioid overdose events to determine the feasibility of using the PCC for automated surveillance. MAIN OUTCOME MEASURE: Quality and availability of required concepts to operationalize metrics and case definitions using PCC data. RESULTS: A subset of the probable case definition may be used for automated surveillance with available structured PCC data. In contrast, logic for confirmed, suspected, and part of the probable case definitions requires additional structured data or analysis of narrative text, which may not contain needed concepts. For example, the confirmed case definition currently requires evidence from narrative text of laboratory confirmation of an opioid in a clinical specimen or diagnosis of opioid overdose in a health care record. CONCLUSION: PCC data are a timely and potentially useful source for automated surveillance of a subset of opioid overdose events, but additional structured and/or coded data are required.

Evaluation in Life Cycle of Information Technology (ELICIT) framework: Supporting the innovation life cycle from business case assessment to summative evaluation.

OBJECTIVE: Our objective was to develop an evaluation framework for electronic health record (EHR)-integrated innovations to support evaluation activities at each of four information technology (IT) life cycle phases: planning, development, implementation, and operation. METHODS: The evaluation framework was developed based on a review of existing evaluation frameworks from health informatics and other domains (human factors engineering, software engineering, and social sciences); expert consensus; and real-world testing in multiple EHR-integrated innovation studies. RESULTS: The resulting Evaluation in Life Cycle of IT (ELICIT) framework covers four IT life cycle phases and three measure levels (society, user, and IT). The ELICIT framework recommends 12 evaluation steps: (1) business case assessment; (2) stakeholder requirements gathering; (3) technical requirements gathering; (4) technical acceptability assessment; (5) user acceptability assessment; (6) social acceptability assessment; (7) social implementation assessment; (8) initial user satisfaction assessment; (9) technical implementation assessment; (10) technical portability assessment; (11) long-term user satisfaction assessment; and (12) social outcomes assessment. DISCUSSION: Effective evaluation requires a shared understanding and collaboration across disciplines throughout the entire IT life cycle. In contrast with previous evaluation frameworks, the ELICIT framework focuses on all phases of the IT life cycle across the society, user, and IT levels. Institutions seeking to establish evaluation programs for EHR-integrated innovations could use our framework to create such shared understanding and justify the need to invest in evaluation. CONCLUSION: As health care undergoes a digital transformation, it will be critical for EHR-integrated innovations to be systematically evaluated. The ELICIT framework can facilitate these evaluations.

A Conceptual Framework of Data Readiness: The Contextual Intersection of Quality, Availability, Interoperability, and Provenance.

BACKGROUND: Data readiness is a concept often used when referring to health information technology applications in the informatics disciplines, but it is not clearly defined in the literature. To avoid misinterpretations in research and implementation, a formal definition should be developed. OBJECTIVES: The objective of this research is to provide a conceptual definition and framework for the term data readiness that can be used to guide research and development related to data-based applications in health care. METHODS: PubMed, the National Institutes of Health RePORTER, Scopus, the Cochrane Library, and Duke University Library databases for business and information sciences were queried for formal mentions of the term "data readiness." Manuscripts found in the search were reviewed, and relevant information was extracted, evaluated, and assimilated into a framework for data readiness. RESULTS: Of the 264 manuscripts found in the database searches, 20 were included in the final synthesis to define data readiness. In these 20 manuscripts, the term data readiness was revealed to encompass the constructs of data quality, data availability, interoperability, and data provenance. DISCUSSION: Based upon our review of the literature, we define data readiness as the application-specific intersection of data quality, data availability, interoperability, and data provenance. While these concepts are not new, the combination of these factors in a novel data readiness model may help guide future informatics research and implementation science. CONCLUSION: This analysis provides a definition to guide research and development related to data-based applications in health care. Future work should be done to validate this definition, and to apply the components of data readiness to real-world applications so that specific metrics may be developed and disseminated.

Developing a sampling method and preliminary taxonomy for classifying COVID-19 public health guidance for healthcare organizations and the general public.

BACKGROUND: Development and dissemination of public health (PH) guidance to healthcare organizations and the general public (e.g., businesses, schools, individuals) during emergencies like the COVID-19 pandemic is vital for policy, clinical, and public decision-making. Yet, the rapidly evolving nature of these events poses significant challenges for guidance development and dissemination strategies predicated on well-understood concepts and clearly defined access and distribution pathways. Taxonomies are an important but underutilized tool for guidance authoring, dissemination and updating in such dynamic scenarios. OBJECTIVE: To design a rapid, semi-automated method for sampling and developing a PH guidance taxonomy using widely available Web crawling tools and streamlined manual content analysis. METHODS: Iterative samples of guidance documents were taken from four state PH agency websites, the US Center for Disease Control and Prevention, and the World Health Organization. Documents were used to derive and refine a preliminary taxonomy of COVID-19 PH guidance via content analysis. RESULTS: Eight iterations of guidance document sampling and taxonomy revisions were performed, with a final corpus of 226 documents. The preliminary taxonomy contains 110 branches distributed between three major domains: stakeholders (24 branches), settings (25 branches) and topics (61 branches). Thematic saturation measures indicated rapid saturation (≤5% change) for the domains of "stakeholders" and "settings", and "topic"-related branches for clinical decision-making. Branches related to business reopening and economic consequences remained dynamic throughout sampling iterations. CONCLUSION: The PH guidance taxonomy can support public health agencies by aligning guidance development with curation and indexing strategies; supporting targeted dissemination; increasing the speed of updates; and enhancing public-facing guidance repositories and information retrieval tools. Taxonomies are essential to support knowledge management activities during rapidly evolving scenarios such as disease outbreaks and natural disasters.

A thematic analysis to examine the feasibility of EHR-based clinical decision support for implementing Choosing Wisely ® guidelines.

OBJECTIVE: To identify important barriers and facilitators relating to the feasibility of implementing clinical practice guidelines (CPGs) as clinical decision support (CDS). MATERIALS AND METHODS: We conducted a qualitative, thematic analysis of interviews from seven interviews with dyads (one clinical expert and one systems analyst) who discussed the feasibility of implementing 10 Choosing Wisely® guidelines at their institutions. We conducted a content analysis to extract salient themes describing facilitators, challenges, and other feasibility considerations regarding implementing CPGs as CDS. RESULTS: We identified five themes: concern about data quality impacts implementation planning; the availability of data in a computable format is a primary factor for implementation feasibility; customized strategies are needed to mitigate uncertainty and ambiguity when translating CPGs to an electronic health record-based tool; misalignment of expected CDS with pre-existing clinical workflows impact implementation; and individual level factors of end-users must be considered when selecting and implementing CDS tools. DISCUSSION: The themes reveal several considerations for CPG as CDS implementations regarding data quality, knowledge representation, and sociotechnical issues. Guideline authors should be aware that using CDS to implement CPGs is becoming increasingly popular and should consider providing clear guidelines to aid implementation. The complex nature of CPG as CDS implementation necessitates a unified effort to overcome these challenges. CONCLUSION: Our analysis highlights the importance of cooperation and co-development of standards, strategies, and infrastructure to address the difficulties of implementing CPGs as CDS. The complex interactions between the concepts revealed in the interviews necessitates the need that such work should not be conducted in silos. We also implore that implementers disseminate their experiences.

Measuring implementation feasibility of clinical decision support alerts for clinical practice recommendations.

OBJECTIVE: The study sought to describe key features of clinical concepts and data required to implement clinical practice recommendations as clinical decision support (CDS) tools in electronic health record systems and to identify recommendation features that predict feasibility of implementation. MATERIALS AND METHODS: Using semistructured interviews, CDS implementers and clinician subject matter experts from 7 academic medical centers rated the feasibility of implementing 10 American College of Emergency Physicians Choosing Wisely Recommendations as electronic health record-embedded CDS and estimated the need for additional data collection. Ratings were combined with objective features of the guidelines to develop a predictive model for technical implementation feasibility. RESULTS: A linear mixed model showed that the need for new data collection was predictive of lower implementation feasibility. The number of clinical concepts in each recommendation, need for historical data, and ambiguity of clinical concepts were not predictive of implementation feasibility. CONCLUSIONS: The availability of data and need for additional data collection are essential to assess the feasibility of CDS implementation. Authors of practice recommendations and guidelines can enable organizations to more rapidly assess data availability and feasibility of implementation by including operational definitions for required data.

State-level adoption of national guidelines for norovirus outbreaks in health care settings.

BACKGROUND: Clinical decision support (CDS) systems can help investigators use best practices when responding to outbreaks, but variation in guidelines between jurisdictions can make such systems hard to develop and implement. This study aimed to identify (1) the extent to which state-level guidelines adhere to national recommendations for norovirus outbreak response in health care settings and (2) the impact of variation between states on outbreak outcomes. METHODS: State guidelines were obtained from Internet searches and direct contact with state public health officials in early 2016. Outcomes from norovirus outbreaks that occurred in 2015 were compared using data from the National Outbreak Reporting System. RESULTS: Guidelines were obtained from 41 of 45 (91%) state health departments that responded to queries or had guidelines available on their Web sites. Most state guidelines addressed each of the national recommendations, but specific guidance varied considerably. For example, among 36 states with guidance on numbers of stool specimens to collect, there were 21 different recommendations. Furthermore, having guidelines consistent with national recommendations was associated with fewer outbreaks reported and more outbreaks with confirmed etiology. CONCLUSIONS: This study identified substantial variation in state health care-associated norovirus outbreak response guidelines, which must be considered when developing related CDS systems. More research is needed to understand why this variation exists, how it impacts outbreak outcomes, and where improvements in evidence-based recommendations and communication of national guidance are needed.

Melinda - A custom search engine that provides access to locally-developed content using the HL7 Infobutton standard.

Healthcare organizations use care pathways to standardize care, but once developed, adoption rates often remain low. One challenge for usage concerns clinicians' difficulty in accessing guidance when it is most needed. Although the HL7 'Infobutton Standard' allows clinicians easier access to external references, access to locally-developed resources often requires clinicians to deviate from their normal electronic health record (EHR) workflow to use another application. To address this gap between internal and external resources, we reviewed the literature and existing practices at the University of Utah Health Care. We identify the requirements to meet the needs of a healthcare enterprise and clinicians, describe the design and development of a prototype to aggregate both internal and external resources from within or outside the EHR, and evaluated strengths and limitations of the prototype. The system is functional but not implemented in a live EHR environment. We suggest next steps and enhancements.

Value Driven Outcomes (VDO): a pragmatic, modular, and extensible software framework for understanding and improving health care costs and outcomes.

OBJECTIVE: To develop expeditiously a pragmatic, modular, and extensible software framework for understanding and improving healthcare value (costs relative to outcomes). MATERIALS AND METHODS: In 2012, a multidisciplinary team was assembled by the leadership of the University of Utah Health Sciences Center and charged with rapidly developing a pragmatic and actionable analytics framework for understanding and enhancing healthcare value. Based on an analysis of relevant prior work, a value analytics framework known as Value Driven Outcomes (VDO) was developed using an agile methodology. Evaluation consisted of measurement against project objectives, including implementation timeliness, system performance, completeness, accuracy, extensibility, adoption, satisfaction, and the ability to support value improvement. RESULTS: A modular, extensible framework was developed to allocate clinical care costs to individual patient encounters. For example, labor costs in a hospital unit are allocated to patients based on the hours they spent in the unit; actual medication acquisition costs are allocated to patients based on utilization; and radiology costs are allocated based on the minutes required for study performance. Relevant process and outcome measures are also available. A visualization layer facilitates the identification of value improvement opportunities, such as high-volume, high-cost case types with high variability in costs across providers. Initial implementation was completed within 6 months, and all project objectives were fulfilled. The framework has been improved iteratively and is now a foundational tool for delivering high-value care. CONCLUSIONS: The framework described can be expeditiously implemented to provide a pragmatic, modular, and extensible approach to understanding and improving healthcare value.

Design, Development, and Initial Evaluation of a Terminology for Clinical Decision Support and Electronic Clinical Quality Measurement.

When coupled with a common information model, a common terminology for clinical decision support (CDS) and electronic clinical quality measurement (eCQM) could greatly facilitate the distributed development and sharing of CDS and eCQM knowledge resources. To enable such scalable knowledge authoring and sharing, we systematically developed an extensible and standards-based terminology for CDS and eCQM in the context of the HL7 Virtual Medical Record (vMR) information model. The development of this terminology entailed three steps: (1) systematic, physician-curated concept identification from sources such as the Health Information Technology Standards Panel (HITSP) and the SNOMED-CT CORE problem list; (2) concept de-duplication leveraging the Unified Medical Language System (UMLS) MetaMap and Metathesaurus; and (3) systematic concept naming using standard terminologies and heuristic algorithms. This process generated 3,046 concepts spanning 68 domains. Evaluation against representative CDS and eCQM resources revealed approximately 50-70% concept coverage, indicating the need for continued expansion of the terminology.

Challenges and Solutions in Optimizing Execution Performance of a Clinical Decision Support-Based Quality Measurement (CDS-QM) Framework.

Given the close relationship between clinical decision support (CDS) and quality measurement (QM), it has been proposed that a standards-based CDS Web service could be leveraged to enable QM. Benefits of such a CDS-QM framework include semantic consistency and implementation efficiency. However, earlier research has identified execution performance as a critical barrier when CDS-QM is applied to large populations. Here, we describe challenges encountered and solutions devised to optimize CDS-QM execution performance. Through these optimizations, the CDS-QM execution time was optimized approximately three orders of magnitude, such that approximately 370,000 patient records can now be evaluated for 22 quality measure groups in less than 5 hours (approximately 2 milliseconds per measure group per patient). Several key optimization methods were identified, with the most impact achieved through population-based retrieval of relevant data, multi-step data staging, and parallel processing. These optimizations have enabled CDS-QM to be operationally deployed at an enterprise level.

Longitudinal Analysis of Computerized Alerts for Laboratory Monitoring of Post-liver Transplant Immunosuppressive Care.

Post-liver transplant patients require lifelong immunosuppressive care and monitoring. Computerized alerts can aid laboratory monitoring, but it is unknown how the distribution of alerts changes over time. We describe the changes over time of the distribution of computerized alerts for laboratory monitoring of post-liver transplant immunosuppressive care. Data were collected for post-liver transplant patients transplanted and managed at Intermountain Healthcare between 2005 and 2012. Alerts were analyzed based on year triggered, time since transplantation, hospitalization status, alert type, action taken (accepted or rejected), reason given for the action taken, and narrative comments. Alerts for overdue laboratory testing became more prevalent as time since transplantation increased. There is an increased need to support monitoring for overdue laboratory testing as the time since transplantation increases. Alerts should support providers as they monitor the evolving needs of post-transplant patients over time. We identify opportunities for improving laboratory monitoring of post-liver transplant patients.

Evaluation of need for ontologies to manage domain content for the Reportable Conditions Knowledge Management System.

The Reportable Condition Knowledge Management System (RCKMS) is envisioned to be a single, comprehensive, authoritative, real-time portal to author, view and access computable information about reportable conditions. The system is designed for use by hospitals, laboratories, health information exchanges, and providers to meet public health reporting requirements. The RCKMS Knowledge Representation Workgroup was tasked to explore the need for ontologies to support RCKMS functionality. The workgroup reviewed relevant projects and defined criteria to evaluate candidate knowledge domain areas for ontology development. The use of ontologies is justified for this project to unify the semantics used to describe similar reportable events and concepts between different jurisdictions and over time, to aid data integration, and to manage large, unwieldy datasets that evolve, and are sometimes externally managed.

Clinical Decision Support-based Quality Measurement (CDS-QM) Framework: Prototype Implementation, Evaluation, and Future Directions.

Electronic quality measurement (QM) and clinical decision support (CDS) are closely related but are typically implemented independently, resulting in significant duplication of effort. While it seems intuitive that technical approaches could be re-used across these two related use cases, such reuse is seldom reported in the literature, especially for standards-based approaches. Therefore, we evaluated the feasibility of using a standards-based CDS framework aligned with anticipated EHR certification criteria to implement electronic QM. The CDS-QM framework was used to automate a complex national quality measure (SCIP-VTE-2) at an academic healthcare system which had previously relied on time-consuming manual chart abstractions. Compared with 305 manually-reviewed reference cases, the recall of automated measurement was 100%. The precision was 96.3% (CI:92.6%-98.5%) for ascertaining the denominator and 96.2% (CI:92.3%-98.4%) for the numerator. We therefore validated that a standards-based CDS-QM framework can successfully enable automated QM, and we identified benefits and challenges with this approach.

Point-of-Care Testing as an Influenza Surveillance Tool: Methodology and Lessons Learned from Implementation.

Objectives. Disease surveillance combines data collection and analysis with dissemination of findings to decision makers. The timeliness of these activities affects the ability to implement preventive measures. Influenza surveillance has traditionally been hampered by delays in both data collection and dissemination. Methods. We used statistical process control (SPC) to evaluate the daily percentage of outpatient visits with a positive point-of-care (POC) influenza test in the University of Utah Primary Care Research Network. Results. Retrospectively, POC testing generated an alert in each of 4 seasons (2004-2008, median 16 days before epidemic onset), suggesting that email notification of clinicians would be 9 days earlier than surveillance alerts posted to the Utah Department of Health website. In the 2008-09 season, the algorithm generated a real-time alert 19 days before epidemic onset. Clinicians in 4 intervention clinics received email notification of the alert within 4 days. Compared with clinicians in 6 control clinics, intervention clinicians were 40% more likely to perform rapid testing (P = 0.105) and twice as likely to vaccinate for seasonal influenza (P = 0.104) after notification. Conclusions. Email notification of SPC-generated alerts provided significantly earlier notification of the epidemic onset than traditional surveillance. Clinician preventive behavior was not significantly different in intervention clinics.

Comparison of computerized surveillance and manual chart review for adverse events.

OBJECTIVE: To understand how the source of information affects different adverse event (AE) surveillance methods. DESIGN: Retrospective analysis of inpatient adverse drug events (ADEs) and hospital-associated infections (HAIs) detected by either a computerized surveillance system (CSS) or manual chart review (MCR). MEASUREMENT: Descriptive analysis of events detected using the two methods by type of AE, type of information about the AE, and sources of the information. RESULTS: CSS detected more HAIs than MCR (92% vs 34%); however, a similar number of ADEs was detected by both systems (52% vs 51%). The agreement between systems was greater for HAIs than ADEs (26% vs 3%). The CSS missed events that did not have information in coded format or that were described only in physician narratives. The MCR detected events missed by CSS using information in physician narratives. Discharge summaries were more likely to contain information about AEs than any other type of physician narrative, followed by emergency department reports for HAIs and general consult notes for ADEs. Some ADEs found by MCR were detected by CSS but not verified by a clinician. LIMITATIONS: Inability to distinguish between CSS false positives and suspected AEs for cases in which the clinician did not document their assessment in the CSS. CONCLUSION: The effect that information source has on different surveillance methods depends on the type of AE. Integrating information from physician narratives with CSS using natural language processing would improve the detection of ADEs more than HAIs.